U.S. Patent Application Publication No. 201408632 A1 discloses a surgical controlling system that includes an endoscope providing a real-time image of surgical environment of a human body, and a processing means. The processing means is adapted to process the real-time image to determine positions of elements (i.e., two surgical instruments) within the real-time image. According the positions of the two surgical instruments, the surgical controlling system can automatically move the endoscope to a position such that a central point of the real-time image is at a middle point between the two surgical instruments. The distance between the two surgical instruments may vary frequently during a surgical operation, so the endoscope may be moved frequently. However, some of the moves of the endoscope are undesirable and may disrupt the surgical operation.